Object detection device

ABSTRACT

Provided is an object detection device capable of reliably recognizing a vehicle. This object detection device detects multiple three-dimensional objects from a left image and a right image captured with a left imaging unit and a right imaging unit (S 103 ), extracts, as combination candidates from among the multiple three-dimensional objects, two three-dimensional objects which exist with an interval to the left/right, and determines whether a sparse parallax region, which is a region having a smaller parallax density than left/right regions, exists in an intermediate region between the two three-dimensional objects extracted as combination candidates. Then, the regions of two three-dimensional objects (for which it has been determined that a sparse parallax region exists in the intermediate region, and which have tentatively been identified as a single three-dimensional object) are extracted respectively from the left and right images and compared to each other, a determination is made regarding whether the perspective is the same, and when it is determined that the perspective is the same, the two three-dimensional objects are determined to be a single three-dimensional object.

TECHNICAL FIELD

The present invention relates to an object detection device.

BACKGROUND ART

One of the main applications of an object detection device is, forexample, to detect an obstacle in the state of being mounted on avehicle. When the obstacle ahead is a vehicle, it is desirable to moreaccurately detect the vehicle and calculate a position and speedinformation thereof. The calculated position and speed information isused, for example, as an input of a collision avoidance function or apreceding vehicle following function, which leads to more appropriatevehicle control.

A background art in this technical field includes JP 2010-224936 A (PTL1). This publication describes that “an object detection device, whichis capable of accurately detecting an object by accurately groupingdistance data detected by a distance detection device, is provided”. PTL1 describes that it is possible to accurately calculate position andspeed information by accurately calculating a region where a vehicleexists on an image using back lamp information of the vehicle.

CITATION LIST Patent Literature

PTL 1: JP 2010-224936 A

SUMMARY OF INVENTION Technical Problem

However, when the lamp information is used to properly detect thevehicle as in PTL 1, the detection performance in a scene where a lampof a preceding vehicle is turned on, such as night time or the middle ofbraking, is improved, but there is no effect in a scene where the lampis turned off. To use characteristics of a vehicle that can be usedregardless of the scene becomes important task, in order to properlydetect the vehicle in more scenes.

The present invention has been made in view of the above-describedpoints, and an object thereof is to provide an object detection devicecapable of stably recognizing a vehicle.

Solution to Problem

An object detection device according to the present invention to solvethe above-described problem includes: a three-dimensional objectdetection unit that detects a plurality of three-dimensional objectsfrom a left image and a right image imaged by a left imaging unit and aright imaging unit; a combination candidate extraction unit thatextracts two three-dimensional objects existing at an interval to theleft and right from among the plurality of three-dimensional objects asa combination candidate; a sparse region determination unit thatdetermines whether or not there is a sparse parallax region in anintermediate region between the two three-dimensional objects extractedas the combination candidate, the sparse parallax region having aparallax density smaller than that of left and right regions; a matchingunit that extracts regions, obtained at the time of assuming the twothree-dimensional objects for which it is determined that the sparseparallax region exists in the intermediate region as onethree-dimensional object, from the left and right images, respectively,and compares the regions with each other to determine whetherperspectives thereof are equal; and a three-dimensional object combiningunit that determines the two three-dimensional objects as onethree-dimensional object when the matching unit determines that theperspectives are equal.

Advantageous Effects of Invention

According to the present invention, it is possible to provide an objectdetection device capable of stably recognizing a vehicle. Incidentally,other objects, configurations, and effects will be apparent from thefollowing description of embodiments.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram for describing an overall configuration of an objectdetection device according to the present embodiment.

FIG. 2 is a configuration diagram of an image processing unit accordingto a first embodiment that performs three-dimensional object combiningprocess.

FIG. 3 is a view illustrating a processing flow of overall processing.

FIG. 4 is a view illustrating a processing flow of a three-dimensionalobject combining unit.

FIG. 5 is a view for describing examples of a case where calculation ofparallax is easy and a case where the calculation of parallax isdifficult.

FIG. 6 is an explanatory view of processing content of a sizedetermination process S221.

FIG. 7 is an explanatory view of processing content of a distancedetermination process S222.

FIG. 8 is a flowchart illustrating details of processing content in asparse parallax region determination process S223.

FIG. 9 is an explanatory view of processing content of a sparse regiondetermination process S304.

FIG. 10 is an explanatory view of processing content of a left and rightmatching process S224.

FIG. 11 is a diagram illustrating a configuration of an image processingunit according to a second embodiment.

FIG. 12 is a view illustrating a processing flow of a three-dimensionalobject division unit 301.

FIG. 13 is a diagram illustrating a configuration of an image processingunit according to a third embodiment.

FIG. 14 is a view illustrating a processing flow of an image processingunit according to the third embodiment.

FIG. 15 is a diagram illustrating a configuration of an image processingunit according to a fourth embodiment.

FIG. 16 is a view illustrating a processing flow according to the fourthembodiment.

FIG. 17 is a diagram illustrating an overall configuration in the caseof including vehicle control according to a fifth embodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present invention will be described withreference to the drawings.

First Embodiment

FIG. 1 is a diagram for describing an overall configuration of an objectdetection device according to the present embodiment. The objectdetection device is mounted on a vehicle and includes a stereo camera100, a memory 103, a CPU 104, an image processing unit 111, and anexternal communication unit 112 as illustrated in FIG. 1. The stereocamera 100 includes a left imaging unit 101 and a right imaging unit 102that image the front of the vehicle as a left and right pair.

The image processing unit 111 extracts a three-dimensional object froman image imaged by the stereo camera 100, tracks the extractedthree-dimensional object in a time-series order, and recognizes whetheror not the object is likely to be a vehicle using a contour or aparallax shape of the tracked three-dimensional object. A recognitionresult is transmitted to the outside of the object detection device bythe external communication unit 112 and used for vehicle control for anaccelerator, a brake, a steering, or the like.

FIG. 2 is a diagram illustrating a configuration of the image processingunit 111. The image processing unit 111 includes a parallax calculationunit 201, a three-dimensional object detection unit 202, athree-dimensional object combining unit 203, and a vehicle recognitionunit 204. The parallax calculation unit 201 calculates parallax which isa deviation in imaging position of the same object using an image imagedby the left imaging unit 101 and an image imaged by the right imagingunit 102. The three-dimensional object detection unit 202 detects athree-dimensional object based on information of the parallax calculatedby the parallax calculation unit 201.

The three-dimensional object detection unit 202 first calculatescoordinates of points in the real space from the parallax informationobtained by the parallax calculation unit 201 and an image coordinate ofthe parallax information in order to detect the three-dimensionalobject. Then, a distance between the points is calculated for therespective points, and points close to each other are grouped. When agrouping result is one three-dimensional object, a three-dimensionalobject considered, as one mass is detected by dividing the object at aplace where the inclination suddenly changes or a region where almost noparallax is obtained. At this time, there is a tendency that the leftand right of a vehicle are detected as different three-dimensionalobjects in the case of detecting the vehicle as the three-dimensionalobject.

FIG. 5 is a view for describing examples of a case where the calculationof parallax is easy and a case where the calculation of parallax isdifficult. For example, when a certain region of a left image isdesignated, whether or not there is a region where the same object isimaged is present in a right image is searched toward a lateraldirection in the right image, for the calculation of parallax.Therefore, in a region where there is a change in brightness in thelateral direction and there is an edge in a vertical direction asillustrated in FIG. 5(1), it is easy to set a corresponding region andto calculate parallax. On the other hand, when there is no change inbrightness in the lateral direction and, for example, an edge in thelateral direction continues as illustrated in FIG. 5(2), it is difficultto set a corresponding region and to calculate parallax.

In the case of the vehicle, it is easy to calculate parallax for leftand right ends since there is an edge in the longitudinal direction, butan edge in the lateral direction is continuous inmost of the centralportion thereof. Therefore, the parallax becomes dense in the left andright parts of the vehicle, and the parallax becomes sparse in thecentral portion. As a result, the left and right ends of the vehicle areoften detected as separate three-dimensional objects.

The three-dimensional object combining unit 203 performs a process ofcombining excessively divided three-dimensional objects. That is, whenit is determined that there is a high possibility that onethree-dimensional object is detected as a plurality of three-dimensionalobjects by the three-dimensional object detection unit 202, thethree-dimensional object combining unit 203 performs the process ofcombining these three-dimensional objects and detecting the combinedobject as one three-dimensional object. Details of the process performedby the three-dimensional object combining unit 203 will be describedlater.

The vehicle recognition unit 204 performs vehicle recognitiondetermination on the three-dimensional object, which has been detectedas the one three-dimensional object by the three-dimensional objectdetection unit 202 and the three-dimensional object combining unit 203,and determines whether or not the object is the vehicle. Even in a casewhere one vehicle is regarded as a plurality of three-dimensionalobjects as the three-dimensional objects detected by thethree-dimensional object detection unit 202, it is possible toappropriately recognize the object as the vehicle in the vehiclerecognition unit 204 since the three-dimensional objects are combined asthe one three-dimensional object in the three-dimensional objectcombining unit 203.

FIG. 3 is a view illustrating a processing flow of overall processing.As illustrated in FIG. 3, first, an image is imaged by the stereo camera100 (S101), and the parallax is calculated by the parallax calculationunit 201 based on the imaged stereo image. Then, the three-dimensionalobject is detected based on information on the calculated parallax(S103). Here, it is determined whether or not one three-dimensionalobject is erroneously detected as a plurality of three-dimensionalobjects, in practice, and the plurality of three-dimensional objects arecombined into the one three-dimensional object (combinedthree-dimensional object) when there is a high possibility of theerroneous detection as the plurality of three-dimensional objects(S104). Then, the vehicle recognition determination on whether or not asingle three-dimensional object detected as one three-dimensional objectin S103 or a combined three-dimensional object combined as onethree-dimensional object in S104 is the vehicle is performed (S105), andthe vehicle control is performed based on a result of the determination(S106).

Next, processing content in the three-dimensional object combining unit203 will be described in detail.

FIG. 4 is a view illustrating a processing flow of the three-dimensionalobject combining unit 203. Data necessary for processing is received ina data input process S210. A three-dimensional object detection resultdetected by the three-dimensional object detection unit 202 is receivedin a three-dimensional object detection result input process S211. Theleft and right images imaged by the left imaging unit 101 and the rightimaging unit 102 are received in a left and right image input processS212.

Next, combination availability determination on whether or notcombination processing may be performed on the receivedthree-dimensional object is performed in a combination availabilitydetermination process S220. At this time, the determination is performedfor a pair of adjacent three-dimensional objects on the image. Herein,the adjacency does not necessarily indicate a state wherethree-dimensional objects are in contact with each other on the image,but indicates a state where the selected two three-dimensional objectsexist at an interval to the right and left without any otherthree-dimensional object (any other three-dimensional object beingdetected) in a region therebetween. In the combination availabilitydetermination process S220, a size determination process S221, adistance determination process S222, a sparse parallax regiondetermination process S223, and a left and right matching process S224are performed.

Next, the combination processing is performed on a pair ofthree-dimensional objects, which have been determined to be combinablein the combination availability determination process S220, in athree-dimensional object combination process S230. For example, aprocess of recalculating a distance between the three-dimensionalobjects by recalculating an average value of parallax included in theregion or updating information on a height and a width that have beenchanged by the combination is performed.

FIG. 6 is an explanatory view of processing content of the sizedetermination process S221. When it is assumed that the pair ofextracted three-dimensional objects is combined, it is determinedwhether or not an aspect ratio and the height and width indicate a sizethat is likely to be the vehicle, which is a detection target (sizedetermination unit).

For example, as illustrated in (1) Three-Dimensional Object DetectionResult, three-dimensional objects adjacent to each other are extractedas a three-dimensional object combination candidate for each ofthree-dimensional objects D1 to D6 (a combination candidate extractionunit). In the example illustrated in (2), a pair of thethree-dimensional object (pedestrian) D1 and the three-dimensionalobject (utility pole) D2 that are adjacent to each other is defined as athree-dimensional object combination candidate (A), a pair of thethree-dimensional objects (pedestrians) D5 and D6 is set as athree-dimensional object combination candidate (C) and a pair of thethree-dimensional objects (left end and right ends of the vehicle) D3and D4 is set as a three-dimensional object combination candidate (B).Such extraction of the three-dimensional object combination candidate isperformed for the entire three-dimensional object in the image, and forexample, a pair of the three-dimensional object (utility pole) D2 andthe three-dimensional object (left end of the vehicle) D3 and a pair ofthe three-dimensional object (right end of the vehicle) D4 and thethree-dimensional object (pedestrian) D5 are also extracted asthree-dimensional object combination candidates although notillustrated.

Then, size determination is performed for each combination candidate(S221) as illustrated in (2) Three-Dimensional Object CombinationCandidate. Here, it is determined whether or not a size obtained in acase where detection regions of two three-dimensional objects are set toboth ends of a predetermined detection target corresponds to a size of adetection target that is decided based on an imaging distance.

For example, the three-dimensional object combination candidate (A),which is the pair of the three-dimensional object (pedestrian) D1 andthe three-dimensional object (utility pole) D2, has a height of theutility pole that is obviously higher than a general vehicle. Thus, theheight is determined as a height that is unlikely to be the vehicle sothat a size determination result is set to NG. Then, thethree-dimensional object combination candidate (C), which is the pair ofthe separated three-dimensional objects (pedestrians) D5 and D6, isdetermined to be likely to be the vehicle in terms of the height, buthas a width that is extremely larger than that of the vehicle. Thus, thewidth is determined as a width that is unlikely to be the vehicle sothat a size determination result is set to NG. Then, thethree-dimensional object combination candidate (B), which is the pair ofthe three-dimensional objects (the right end and the left end of thevehicle) D3 and D4 is determined such that both of a height and a widthare likely to be the vehicle so that a size determination result is setto OK.

FIG. 7 is an explanatory view of processing content of the distancedetermination process S222. When it is assumed the pair of extractedthree-dimensional objects is combined, it is determined whether or notdistance information between the three-dimensional objects as the pairhas similar values (distance determination unit).

For example, as illustrated in (1) Three-Dimensional Object DetectionResult, three-dimensional objects adjacent to each other are extractedas a three-dimensional object combination candidate for each ofthree-dimensional objects D7 to D12 (combination candidate extractionunit). In the example illustrated in FIG. 7, a pair of thethree-dimensional object (pedestrian) D7 and the three-dimensionalobject (utility pole) D8 that are adjacent to each other is defined as athree-dimensional object combination candidate (D) (a pair of thethree-dimensional objects (pedestrians) D11 and D12 is set as athree-dimensional object combination candidate (F), and a pair of thethree-dimensional objects (left end and right ends of the vehicle) D9and D10 is set as a three-dimensional object combination candidate (E).Although not illustrated, a pair of the three-dimensional object(utility pole) D8 and the three-dimensional object (left end of thevehicle) D9 and a pair of the three-dimensional object (right end of thevehicle) D10 and the three-dimensional object (pedestrian) D11 are alsoextracted as three-dimensional object combination candidates.

Then, distance determination is performed for each of the combinationcandidates (D) to (F) (S222) as illustrated in (2) Three-DimensionalObject Combination Candidate and (3) Overhead View. For example,although the combination candidate (D), which is the pair of thethree-dimensional object (pedestrian) D7 and the three-dimensionalobject (utility pole) D8, is taken as the three-dimensional combinationcandidate since the objects are adjacent to each other on the image, theutility pole D8 is disposed at a position apart while the pedestrian D7is disposed at the front so that the distance information on imagingdistance is greatly deviated between the pair, which is set to distancedetermination NG. Similarly, the distance information on imagingdistance is greatly deviated also for the distance candidate (F), whichis the pair of the three-dimensional objects (pedestrians) D11 and D12,and thus, the distance determination NG is set. On the other hand, thedistance information on imaging distance is substantially the same forthe combination candidate (E), which is the pair of thethree-dimensional object (the right end and the left end of the vehicle)D3 and D10, and thus, a distance determination result is set to OK.

When the determination NG is set for at least any one of the sizedetermination process S221 and the distance determination process S222,the relevant three-dimensional object pair is excluded from thedetermination, and the processing is terminated as combinationdetermination NG at such a point in time. This is for mitigation ofprocessing unavailability of the entire three-dimensional objectcombining unit 203 by narrowing down a candidate in advance because theprocessing in the left-right matching process S224 is highly likely tobe unavailable. Then, when the determination OK are obtained in both thesize determination process S221 and the distance determination processS222, the relevant three-dimensional object pair is considered as adetermination target of the subsequent sparse parallax regiondetermination process S223.

In the sparse parallax region determination process S223, it isdetermined whether or not there is a sparse parallax region of whichparallax density is lower than that of left and right region in anintermediate region between the pair of three-dimensional objects.

FIG. 8 is a flowchart illustrating details of processing content in thesparse parallax region determination process S223. Information on athree-dimensional object pair, obtained as a determination target ofsparse region determination by the processing of the size determinationprocess S221 and the distance determination process S222, is received ina three-dimensional object pair information input process S301. In adetermination region decision process S302, the left and right regionswhere the three-dimensional object pair exists and the intermediateregion therebetween are extracted, and the intermediate region is set asthe target of the sparse region determination. In a sparse determinationthreshold calculation process S303, a parallax density threshold Th1,configured to determine whether or not the intermediate region is asparse parallax region, is calculated and decided. In a sparse regiondetermination process S304, it is determined whether or not theintermediate region between the three-dimensional object pair is thesparse parallax region based on the parallax density threshold Th1.

FIG. 9 is an explanatory view of processing content of the sparse regiondetermination process S304. As a specific process, for example, thedensity of parallax is calculated for each vertical row with respect toleft and right regions A1 and A2 of the three-dimensional object pairand an intermediate region A3 therebetween. The area of valid parallax(a point at which stereo matching is succeeded) existing in a certainvertical row may be divided by the area of one vertical row to calculatethe parallax density. With respect to the obtained density,determination on whether the region is the sparse parallax region or adense region is performed based on the parallax density threshold (firstthreshold) Th1 (sparse region determination unit). It is determined asthe sparse parallax region when the parallax density is equal to orlower than the parallax density threshold Th1, and it is determined asthe parallax dense region when the parallax density is higher than theparallax density threshold Th1.

The parallax density threshold Th1 is calculated in a sparsedetermination threshold calculation process S303. An appropriate fixedvalue may be set, or a ratio with respect to the left and right parallaxdense regions may be set and made variable. In the drawing, an exampleof the parallax density and the determination result when the parallaxdensity threshold Th1 is fixed is illustrated. Although the technique ofcalculating the density for each vertical row of the image has beenillustrated as an example in the present embodiment, a region may bedivided into regions (left and right) detected as three-dimensionalobjects and an intermediate region therebetween, and each density of theregions may be calculated.

FIG. 10 is an explanatory view of processing content of the left-rightmatching process S224.

A region obtained at the time of assuming combination is extracted fromeach of the left and right images with respect to the three-dimensionalobject pair, which is the combination candidate and of which theparallax density of the intermediate region has been determined to be asparse parallax region, and matching determination on whether or notperspectives are the same by comparing the regions with each other isperformed (matching unit). A correlation value such as a SAD is used forthe matching determination. The SAD is obtained by taking a differencefor each coordinate in a block and adding absolute values of thedifferences. When the correlation value is equal to or smaller than acorrelation value threshold (second threshold) Th2, it is determinedthat the perspective of the extracted region is the same between theleft and right images. When it is determined that the perspective is thesame, the three-dimensional object pair, which is the combinationcandidate, is determined as one three-dimensional object (combinedthree-dimensional object).

When the three-dimensional object pair, which is the combinationcandidate, includes different three-dimensional objects, the perspectivediffers between the left and right images because the background or thelike is imaged at each central portion thereof, and the SAD takes alarge value. Therefore, when the correlation value is larger than thecorrelation value threshold Th2, it is determined that the perspectiveof the extracted image is not the same between the left and rightimages. When it is determined that the perspective of the extractedimage differs between the left and right images in this manner, thethree-dimensional object pair, which is the combination candidate, isdetermined as the plurality of three-dimensional objects, that is, twoindependent three-dimensional objects.

For example, in the case of a combined three-dimensional objectcandidate (vehicle) GL1 of a left image and a combined three-dimensionalobject candidate (vehicle) GR1 of a right image, a correlation value isequal to or smaller them the correlation value threshold Th2, and it isdetermined that the perspective is the same between the left and rightimages. Therefore, the three-dimensional object pair is determined asone combined three-dimensional object, and it is determined that thethree-dimensional object can be combined (combination OK).

On the other hand, in the case of a combined three-dimensional objectcandidate (pedestrian pair) GL2 of a left image and a combinedthree-dimensional object candidate (pedestrian pair) GR2 of a rightimage, the perspective of the central portion of the image greatlydiffers between the left and right images, and thus, it is determinedthat a correlation value is larger than the correlation value thresholdTh2, and the perspective differs between the left and right images.Therefore, the three-dimensional object pair is determined as aplurality of three-dimensional objects, and it is determined that thethree-dimensional objects are not allowed to be combined (combinationNG).

Three-dimensional object pairs determined to be combinable by the abovedetermination are integrated as one three-dimensional object (combinedthree-dimensional object) by three-dimensional object combinationprocess S230 in FIG. 4. The integrated combined three-dimensional objectis input to the vehicle recognition unit 204 of FIG. 2, and it isdetermined whether or not the object is the vehicle. When beingdetermined as the vehicle, the object is properly recognized as thevehicle. A vehicle recognition result is transmitted to the outside bythe external communication unit 112 in FIG. 1, and is used forapplications such as the vehicle control.

According to the object detection device of the present embodiment, thethree-dimensional object pair, which includes the sparse parallax regionhaving the parallax density equal to or lower than the threshold Th1 inthe intermediate region of the three-dimensional object pair, is set asthe combination candidate, the matching determination is performed byextracting an image of the region obtained at the time of combining thethree-dimensional object pair as the combination candidate from left andright images, and it is determined that the three-dimensional objectpair is the one three-dimensional object when the perspective is thesame between the left and right images, and it is determined that thethree-dimensional object pair is the plurality of three-dimensionalobjects when the perspectives thereof differ from each other. Therefore,it is possible to appropriately detect the vehicle in more scenesregardless of whether or not a lamp of a preceding vehicle is turned on.

Although the case where an obstacle is the vehicle has been described inthe present embodiment, the detection target is not limited to thevehicle, but may be another obstacle, for example, a pedestrian and thelike. According to the present embodiment, the three-dimensional objectpair is not combined when the determination on the size and the imagingdistance of the three-dimensional object pair, and the determination onthe sparse parallax region results is NO, and thus, there is a highprobability that it is possible to accurately detect a three-dimensionalobject having a pedestrian-size which is smaller than a size of thevehicle.

Second Embodiment

Here, illustrated is an embodiment in a case where the present inventionis applied from the viewpoint of appropriately dividing athree-dimensional object in the state of being excessively combinedinstead of appropriately combining the three-dimensional objects in thestate of being excessively divided as illustrated in the firstembodiment.

FIG. 11 is a diagram illustrating a configuration of an image processingunit 111 according to the present embodiment. Constituent elements whichare the same as those of the first embodiment will be denoted by thesame reference signs, and a detailed description thereof will beomitted.

The parallax calculation unit 201 and the vehicle recognition unit 204perform the same processing as that illustrated in the first embodiment.As compared with the first embodiment, the three-dimensional objectdetection unit 202 is practically adjusted so as to easily detect aplurality of three-dimensional objects as one three-dimensional object.For example, coordinates in the real space are calculated from theparallax information and the image coordinates obtained by the parallaxcalculation unit 201, and the points adjacent to each other are grouped,but a grouping threshold at this time is set to be large. Alternatively,when dividing the grouping result, the division in a region whereparallax is not obtained is not performed.

Through such adjustment, it is possible to suppress a case ofexcessively dividing one three-dimensional object such as a vehicle, butit is more likely to erroneously detect a plurality of three-dimensionalobjects such as two pedestrians as one three-dimensional object.Therefore, a process of dividing the excessively combinedthree-dimensional object is performed in a three-dimensional objectdivision unit 301.

FIG. 12 is a view illustrating a processing flow of thethree-dimensional object division unit 301. Data necessary forprocessing is received in a data input process S410. A three-dimensionalobject detection result detected by the three-dimensional objectdetection unit 202 is received in a three-dimensional object detectionresult input process S411. The left and right images imaged by the leftimaging unit 101 and the right imaging unit 102 are received in a leftand right image input process S412.

Next, determination on whether or not division processing may beperformed on the received three-dimensional object is performed in adivision availability determination process S420. For example, a regionwith sparse parallax is searched (division candidate point search unit)in a division candidate point searching process S421 similarly to thesparse parallax region determination process S223. A dense parallaxregion and a sparse parallax region are extracted from a region of animage detected as one three-dimensional object. When the dense parallaxregion exists on the left and right and the sparse parallax regionexists at the center, end portions of the dense parallax regions are setas division candidate points.

Next, a region in which the three-dimensional object is detected isextracted from left and right images, and it is determined whether ornot perspectives tire the same between left and right images in aleft-right matching process S422. A correlation value such as a SAD isused for the determination. When a correlation value is equal to orsmaller than a threshold, it is determined that the left and rightimages are equal. When a value of the SAD is large, it is determined tobe dividable since the perspective differs between the left and rightimages.

The three-dimensional object that has been determined to be dividablethrough the above-described determination is input to athree-dimensional object division process S430, and is divided into aplurality of three-dimensional objects. For example, a process ofrecalculating a distance between the three-dimensional objects bycalculating an average value of parallax included in each of dividedregions or updating information on a height and a width that have beenchanged by the division is performed. Thereafter, the resultant is inputto the vehicle recognition unit 204 similarly to the first embodimentand processed in the same manner as the other three-dimensional objects.

According to the present embodiment, the detection of thethree-dimensional object is roughly performed first, and then, it isdetermined whether or not the object that has been detected as onethree-dimensional object is dividable. Since the determination on thedivision availability is performed only for the three-dimensional objectdetected as one three-dimensional object by the three-dimensional objectdetection in the present embodiment, there are fewer processing targetsas compared to the first embodiment, and it is possible to shorten theprocessing time. The present embodiment is suitable for detection of arelatively larger detection target such as a vehicle than a relativelysmaller detection target such as a pedestrian.

Third Embodiment

A third embodiment is a modification of the first embodiment. Herein,the embodiment in which a threshold used for combination determinationof the three-dimensional object combining unit 203 is dynamicallychanged is illustrated.

Although a width, a height, an aspect ratio, and the like that arelikely to be the vehicle are used for determination in thethree-dimensional object combining unit 203, there are various shapeseven in the vehicle sans phrase. Accordingly, it is desirable that thethreshold can be dynamically changed depending on the target in order toprevent erroneous determination.

FIG. 13 is a diagram illustrating a configuration of the imageprocessing unit ill according to the present embodiment. Constituentelements which are the same as those of the first and second embodimentswill be denoted by the same reference signs, and a detailed descriptionthereof will be omitted.

Each processing content of from the parallax calculation unit 201 to thevehicle recognition unit 204 is the same as that in the firstembodiment. The threshold to be used in the three-dimensional objectcombining unit 203 is calculated and dynamically changed in acombination threshold adjustment unit 401 added in the presentembodiment.

FIG. 14 is a view illustrating a processing flow according to thepresent embodiment. A three-dimensional object detection result in thethree-dimensional object detection process S103 and a vehiclerecognition result in the vehicle recognition process S105 are received,as inputs, in combination determination threshold adjustment processS501. For example, a height threshold in the size determination processS221 is taken as an example. For example, it is assumed thatdetermination on a height is determined as OK when the height fallswithin “height of 1 m to 3 m” set in order to correspond to from acompact car to a large car, as an initial value of the height threshold.

Then, when a three-dimensional object with a height of 1.5 m isrecognized as a vehicle as a result of vehicle recognition by thevehicle recognition process S105, it is possible to set thethree-dimensional object recognized as the vehicle as a combinationtarget and suppress erroneous combination other them the target bychanging the threshold to “height of 1 m to 2 m”.

Although the determination on the height has been taken as an example,it is a matter of course that it is possible to adjust variousthresholds, for example, a width and an aspect ratio in the sizedetermination process S221, a difference threshold (third threshold) ofimaging distance in the distance determination process S222, a thresholdof sparse region determination in the sparse parallax regiondetermination process S223, and the like, depending on a size andvehicle type informal; ion of the vehicle. An obtained vehiclerecognition result is transmitted to the outside by the externalcommunication unit 112, and is used for applications such as the vehiclecontrol, which is similar to the first embodiment.

Fourth Embodiment

A fourth embodiment is a modification of the first embodiment. Here,illustrated is an embodiment in which a target region is extracted inadvance on an image at the time of extracting a three-dimensional objectas a target in the three-dimensional object combining unit 203. It ispossible to narrow down a three-dimensional object as a determinationtarget so that an overall processing load is reduced.

FIG. 15 is a diagram illustrating a configuration of the imageprocessing unit 111 according to the present embodiment. Constituentelements which are the same as those of the first to third embodimentswill be denoted by the same reference signs, and a detailed descriptionthereof will be omitted.

Each processing content of from the parallax calculation unit 201 to thevehicle recognition unit 204 is the same as that in the firstembodiment. A region to be subjected to three-dimensional objectcombination determination is extracted by the three-dimensional objectcombining unit 203 in an added combination determination target regiondetection unit 501.

FIG. 16 is a view illustrating a processing flow according to thepresent embodiment. A parallax image is used as an input to extract aregion as a combination determination target in a combinationdetermination target region detection process S601. For example,embodiments of performing the combination of the three-dimensionalobjects in the first embodiment, and performing the division of thethree-dimensional object in the second embodiment have been illustrated,and both the embodiments are common in terms of performing thecombination or division in the sparse parallax region. Accordingly, aparallax density map is generated for the entire parallax image in thecombination determination target region detection process S601.

Here, a region where the parallax density is sparse and its surroundingregion are extracted as combination determination target regions.Whether or not a region is the region with the sparse parallax densityis determined based on a parallax density threshold (fourth threshold)set in advance, and it is determined as the sparse region when theparallax density on the image is equal to or lower than the parallaxdensity threshold. Only the extracted region is processed as thecombination target in the three-dimensional object combination processS104. Accordingly, there is no need to perform the determination oncombination or division in a region where parallax is sufficientlyobtained, and the processing load is reduced.

Meanwhile, for example, it is possible to consider region extractionusing information on a traveling path of a subject vehicle estimated onthe basis of CAM information and road surface information. The travelingpath of the subject vehicle is estimated from the CAN information suchas speed and a steering angle of the subject vehicle and surroundingenvironment information such as a white line, and only athree-dimensional object positioned on the subject vehicle travelingpath is input into the three-dimensional object combination process S104as a combination target. In the case of using this technique, it ispossible to reduce the processing load particularly without missing theobject on the traveling path relating to vehicle control. An obtainedvehicle recognition result is transmitted to the outside by the externalcommunication unit 112, and is used for applications such as the vehiclecontrol, which is similar to the first embodiment.

Fifth Embodiment

Here, illustrated is an embodiment of a case where the present inventionis applied to a system in which vehicle detection is performed using astereo camera mounted on a vehicle and a detection result iscommunicated with the outside to perform vehicle control.

FIG. 17 is a diagram illustrating the overall configuration in the caseof including the vehicle control. Constituent elements which are thesame as those of the first to fourth embodiments will be denoted by thesame reference signs, and a detailed description thereof will beomitted.

When performing the vehicle control, it is desirable that a controlmethod can be changed depending on detection accuracy. For example, inorder to implement a preceding vehicle following function, anappropriate position and speed are calculated to control the vehiclewhen a detected vehicle is definitely a vehicle, but there is a riskthat an error such as a deviation in detection position is generated inthe opposite case.

Thus, detection reliability information is added to a signal, to betransmitted to the outside in the external communication unit 112, in areliability calculation unit 610. Examples of the reliabilityinformation include information to distinguish the three-dimensionalobject detection result in the three-dimensional object detection unit202 from the detection result in the three-dimensional object combiningunit 203. For example, a flag, configured to distinguish whether thetarget is a combined three-dimensional object combined in thethree-dimensional object combining unit 203 or a singlethree-dimensional object detected by the three-dimensional objectdetection unit 202, is added to an external communication signal.

The signal to the outside is received by a vehicle control unit 600 tocontrol a brake 601, an alarm 602, an accelerator 603, a steering 605,and the like. There is a possibility that a detection result of thevehicle (combined three-dimensional object) detected as the result ofthe three-dimensional object combining unit 203 is unstable as comparedto the case of the vehicle (single three-dimensional object) detected asthe result of the three-dimensional object detection unit 202.

Thus, the normal preceding vehicle following function is implemented,for example, if the vehicle corresponds to the detection result (singlethree-dimensional object) in the three-dimensional object detection unit202. When the vehicle corresponds to the detection result (combinedthree-dimensional object) in the three-dimensional object combining unit203, however, an operation of only weakening the control of the brakeS01 or continuing the control by considering the possibility oftemporarily losing sight due to the fail of combination in thethree-dimensional object combining unit 203 and assuming that there isthe vehicle for a very short time even after losing the sight isconceivable.

Incidentally, the present invention is not limited to theabove-described embodiments, and includes various modification examples.For example, the above-described embodiments have been described indetail in order to describe the present invention in an easilyunderstandable manner, and are not necessarily limited to one includingthe entire configuration that has been described above. In addition,some configurations of a certain embodiment can be substituted byconfigurations of another embodiment, and further, a configuration ofanother embodiment can be also added to a configuration of a certainembodiment. Further, addition, deletion or substitution of otherconfigurations can be made with respect to some configurations of eachembodiment.

In addition, the above-described respective configurations may beconfigured such that some or the whole of them may be implemented by aprocessor that executes a program even when being configured byhardware. In addition, only a control line and an information lineconsidered to be necessary for the description have been illustrated,and all control lines and information lines required as a product arenot illustrated. It may be considered that most of configurations arepractically connected to each other.

REFERENCE SIGNS LIST

101 left imaging unit

102 right imaging unit

111 image processing unit

112 external communication unit

201 parallax calculation unit

202 three-dimensional object detection unit

203 three-dimensional object combining unit

204 vehicle recognition unit

301 three-dimensional object division unit

401 combination threshold adjustment unit

501 combination determination target region detection unit

600 vehicle control unit

The invention claimed is:
 1. An object detection device comprising: athree-dimensional object detection unit that detects a plurality ofthree-dimensional objects from a left image and a right image imaged bya left imaging unit and a right imaging unit; a combination candidateextraction unit that extracts two three-dimensional objects existing atan interval to the left and right from among the plurality ofthree-dimensional objects as a combination candidate; a sparse regiondetermination unit that determines whether or not there is a sparseparallax region in an intermediate region between the twothree-dimensional objects extracted as the combination candidate, thesparse parallax region having a parallax density smaller than that ofleft and right regions; a matching unit that extracts regions, obtainedat the time of assuming the two three-dimensional objects for which itis determined that the sparse parallax region exists in the intermediateregion as one three-dimensional object, from the left and right images,respectively, and compares the regions with each other to determinewhether perspectives thereof are equal; and a three-dimensional objectcombining unit that determines the two three-dimensional objects as theone three-dimensional object when the matching unit determines that theperspectives are equal.
 2. The object detection device according toclaim 1, wherein the sparse region determination unit determines thatthe sparse parallax region exists when the parallax density is equal toor lower than a first threshold.
 3. The object detection deviceaccording to claim 1, wherein the matching unit determines that theperspective is equal when a correlation value obtained by addingabsolute values of differences of the regions extracted from the leftand right images is equal to or smaller than a second threshold.
 4. Theobject detection device according to claim 1, further comprising: a sizedetermination unit that determines whether or not a size obtained whendetection regions of the two three-dimensional objects are set to bothends of a predetermined detection target corresponds to a size of thedetection target determined based on an imaging distance; and a distancedetermination unit that compares imaging distances of the twothree-dimensional objects extracted as the combination candidates anddetermines whether or not the imaging distances are equal to each other,wherein the sparse region determination unit performs determination onwhether or not the sparse parallax region exists when it is determinedto correspond to the size of the detection target by the sizedetermination unit and it is determined that the respective imagingdistances are equal by the distance determination unit.
 5. The objectdetection device according to claim 4, wherein the distancedetermination unit determines that the respective imaging distances areequal when a difference between the respective imaging distances of thetwo three-dimensional objects is equal to or smaller than a thirdthreshold.
 6. The object detection device according to claim 4, furthercomprising: a vehicle recognition unit that recognizes whether or notthe combined three-dimensional object determined as the onethree-dimensional object by the three-dimensional object combining unitis a vehicle; and a combination threshold adjustment unit that adjusts adetermination threshold of the three-dimensional object combining unitaccording to a size or vehicle type information of the vehiclerecognized by the vehicle recognition unit.
 7. The object detectiondevice according to claim 1, further comprising a combinationdetermination target region detection unit that extracts a sparse regionof which parallax density on an image is equal to or lower than a fourththreshold set in advance, as a combination determination target region,wherein the three-dimensional object combining unit performs thedetermination only for the combination determination target region as aprocessing target.
 8. The object detection device according to claim 1,wherein the three-dimensional object combining unit sets only athree-dimensional object on a subject vehicle traveling path, estimatedbased on CAN information and road surface information, as a combinationtarget.
 9. The object detection device according to claim 1, furthercomprising a vehicle control unit, wherein information to distinguish athree-dimensional object detection result obtained by thethree-dimensional object detection unit from a detection result obtainedby the three-dimensional object combining unit is added to an externalcommunication signal.
 10. An object detection device comprising: athree-dimensional object detection unit that detects a three-dimensionalobject from left and right images imaged by left and right imagingunits; a division candidate point search unit that determines whether ornot a region having a sparse parallax density than left and rightregions exists in an intermediate region of the three-dimensional objectand sets end portions of a dense parallax region as division candidatepoints when it is determined that the region exists; a matching unitthat extracts a region from which the three-dimensional object isdetected from the left and right images, and compares the regions witheach other to determine whether or not perspectives are different; and athree-dimensional object division unit that divides thethree-dimensional object at the division candidate points when it isdetermined that the perspectives are different by the matching unit.